Nugget-Based Computation of Graded Relevance

We propose a simple method for assigning graded relevance values to documents judged during the course of a retrieval experiment. In making this proposal, we aim to avoid the potential for ambiguity and greater cognitive load associated with standard graded relevance judgments. Under our proposal, we first decompose a retrieval topic into a number of informational nuggets. For each document, a binary judgment is made with respect to each nugget. The ratio of relevant nuggets to total nuggets becomes the graded relevance value assigned to that document. To provide support for this idea, we turn to test collections created for the TREC Web Track. Along with the usual graded relevance judgments required by traditional effectiveness measures, these test collections include topic decompositions created for the purpose of evaluating novelty and diversity. By exploiting these test collections for our own purposes, we demonstrate a clear relationship between our proposed method and traditional graded relevance. In addition to supporting our proposal, our experiments suggest that informational nuggets can provide a unified approach to relevance assessment, supporting both traditional effectiveness measures and newer measures of novelty and diversity.